User-centric contextual information for browser

ABSTRACT

A method for providing contextual information to a user during a browsing session includes maintaining a user-centric graph including a plurality of user-centric facts associated with the user and derived from interaction by the user with a plurality of different computer services. The method further includes recognizing a context of interaction with a browser application by a user. The method further includes identifying assistive information pertaining to the context, and displaying the assistive information to the user. The assistive information may be based at least on one or more user-centric facts in the user-centric graph.

BACKGROUND

Users frequently use browser software to access information sources(e.g., the Web, computer file systems, databases, libraries, etc.), tocomplete various tasks and engage in various activities. However,typical browsers are not configured to recognize, at a rich level ofgranularity, the context of usage (e.g., details about a current task oractivity) in order to present information pertaining to tasks ofinterest. Instead, typical browsers present information related to acurrent document (e.g., web page, computer file, or database entry)being visited, without particular regard to the context of usage.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

A method for providing contextual information to a user during abrowsing session includes maintaining a user-centric graph including aplurality of user-centric facts associated with the user and derivedfrom interaction by the user with a plurality of different computerservices. The method further includes recognizing a context ofinteraction with a browser application by a user. The method furtherincludes identifying assistive information pertaining to the context,and displaying the assistive information to the user. The assistiveinformation may be based at least on one or more user-centric facts inthe user-centric graph.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows an exemplary computer system for providing user-centriccontextual information in a browser. FIG. 1B shows a further example ofa user context in which a browser provides user-centric contextualinformation.

FIG. 2 shows a method of providing user-centric contextual informationin a browser.

FIGS. 3A-3B show an exemplary user-centric graph data structure forstoring a collection of user-centric facts.

FIG. 4 shows an exemplary computer system.

DETAILED DESCRIPTION

The present disclosure is directed to a method for browser programs toprovide contextual information to a user, e.g., to assist the userduring a browsing session. Non-limiting examples of browser programsinclude desktop, web, mobile phone, and/or other computer servicesconfigured to browse through and access a plurality of content items,e.g., multimedia content, web pages, or any other data. Non-limitingexamples of browsing may include 1) viewing a listing of contentorganized in any suitable manner (e.g., organized by a file location orweb address), 2) searching by content and/or metadata (e.g., by contenttype, by natural language contents, by text strings and/or patterns, byimage content, etc.), 3) accessing files from a listing and/or search(e.g., displaying an image from a search result), and/or 4)accessing/viewing curated, suggested or otherwise retrieved content. Forexample, browser programs include web browsers, filesystem browsers, aswell as browsing software configured for accessing library records,medical records, financial data, news archives, and/or any othersuitable information which may be stored/organized in a browsablemanner.

The present disclosure improves the technology of browser programs,including improvements to browser UI and improvements to browserbackend. In particular, the present disclosure includes a method for abrowser to present a user with assistive information related to acurrent interaction context of the user, which may be useful for variousreasons, such as to assist the user in finding information of interest,engaging in activities of interest, and/or completing tasks using thebrowser. By presenting the user with relevant information that mayfacilitate completion of a task, the efficiency of task completion bythe user and the computational efficiency of the browser may beimproved. For example, the user may be able to complete a task morequickly and/or with fewer interactions with the browser, by utilizingcontextual information presented by the browser in lieu of extensiveadditional searching/browsing. In some examples, the assistiveinformation may be relevant to potential future steps of a task, so theuser may be able to plan ahead and complete the task more efficiently byconsidering the information related to future steps. Similarly, fewercomputational steps and/or less network communication may be required bythe time the user has completed the task, e.g., the browser may be ableto batch together relevant information from multiple different sourcesand efficiently deliver the batched information to the user, in lieu ofthe user making multiple requests to get each individual piece ofinformation. Furthermore, according to the present disclosure,information relating to user interaction with different computerservices may be aggregated in a common user-centric graph, therebyreducing the need for multiple different queries (e.g., associated withdifferent computer services) to access the relevant information. Theaggregation of relevant information from multiple different computerservices may enable an intelligent, automatic presentation of assistiveinformation that the user can utilize in lieu of making multiple manualqueries associated with multiple different computer services. Forexample, if a user's task may be assisted by receiving the top searchresult for each of 3 different web searches, the browser may be able toanticipate that the user would likely conduct each of the 3 web searchesin the near future, in order to present said 3 top search results,instead of waiting for the user to make each of the 3 different websearches and presenting other, less relevant results of each search.FIG. 1A shows an exemplary computing system 100 including a clientcomputer 102. Client computer 102 may be any suitable computing device,e.g., a desktop computer or mobile phone device. Client computer 102 isconfigured to instantiate a browser 110. As shown in FIG. 1A, browser110 may be provided as a GUI program including visual elements forviewing/navigating information sources. When browser 110 is to beprovided as a GUI, client computer 102 may include and/or operativelycouple to one or more display devices configured to visually presentbrowser 110. Although examples are discussed in terms of a browserincluding a GUI for display, in other examples, a browser 110 mayinclude other user interface elements (e.g., text, natural language,speech audio, and/or user interfaces configured for interaction with auser via any other suitable state-of-the-art and/or future inputdevices).

For example, in FIG. 1A, browser 110 is a web browser displaying twocontent tabs 111 (with title indicating a page related to a hotel) and112 (with title indicating a page related to an airline). Althoughexamples are discussed in terms of a web browser, browser 110 mayalternately or additionally include any other kind of browser, e.g., afilesystem browser. Content area 113 shows visual content associatedwith first content tab 111, namely booking information for a hotelincluding a booking date calendar 114 and a “BOOK NOW!” button 115.Browser 110 further includes an assistive information button 120configured to indicate when contextual information is available forassisting the user and selectable to display an information pane 121configured to display such contextual information. Contextualinformation displayed in information pane 121 may include any suitableinformation for assisting the user with a task and/or activity they maybe engaged in. For example, in FIG. 1A, information pane 121 showscontextual information related to the user's travel plans, which mayassist the user with deciding on an available booking date usingcalendar 114. Information pane 121 may additionally or alternately showfurther information not shown in FIG. 1A. For example, when a context isrelated to travelling, information pane 121 may show further informationrelated to future and/or historical trips. In an example, a user mayhave gone to the same location in a previous trip in a previous year.Accordingly, information pane 121 may present relevant information fromthe previous trip. For example, information pane 121 may include anautomatic suggestion to go to a web site associated with a hotel atwhich the user stayed during the previous trip. For example, informationpane 121 may show one or more photos and/or links to photo/video albumsassociated with the user's previous trip.

To provide relevant assistive information to facilitate easiercompletion of the task, computing system 100 intelligently recognizes acontext of user activity. Non-limiting examples of contexts include 1) atask and/or activity (e.g., a current task the user is engaged in, arecent task the user completed, and/or a future task the user hasscheduled and/or made a commitment to perform); 2) the state of beingengaged in an ongoing and/or long-term commitment (e.g., a job, a class,and/or a project) 3) a collection of related tasks, activities, websites, snippets of information, web searches, contact information forcollaborators, documents, etc., relating to a particular endeavor suchas research and development, academic or scholastic studies, etc.;and/or 4) a predefined topic of interest to the user and/or a topicrecognized based on recent, current, and/or historical activity of theuser (e.g., a favorite television show, a current research topic).

Computing system 100 may use personal information of the user arisingfrom user interaction with the browser and/or with any of a plurality ofother computer services, in order to recognize a current context of theuser. For example, the current context of the user may be recognizedbased on a collection of user-centric facts derived from the userinteraction with the browser and/or other computer services. Similarly,the listed assistive information that is provided to a user in responseto a recognized context may be drawn from any combination of userinteractions with the browser and/or other computer services. “Computerservice” may refer herein to software applications executable oncomputer systems (e.g., including networked computing systems, cloudcomputing, and the like), hardware and/or software products, and/or anyother machines or processes configured to provide interactive and/ornon-interactive services to one or more users, in order to assist theuser(s), edit, transform, store, and/or maintain data, interact withother computer services, and the like. The other computer services mayinclude web-based, desktop, and/or mobile phone applications, smart homedevices/“Internet of Things” devices, intelligent assistant devices,wearable devices, background services, etc. Non-limiting examples ofcomputer services from which personal information may be derivedinclude: 1) an intelligent assistant program; 2) an email program; 3) anoffice suite program (e.g., word processor, slideshow, spreadsheetprogram); 4) a calendar program; 5) a teleconferencing program; 6) atask tracking program; 7) a health/activity/nutrition tracking program;8) a multimedia program (e.g., television/video, music, and/or videogame content delivery program); 9) a diagramming/publishing program; 10)an integrated development environment for computer programming; 11) asocial network service; 10) a workplace collaboration environment; and11) a cloud data storage and file synchronization program.

As indicated above, the user browser context and assistive informationprovided in response to that context may be derived from user-centricfacts from various sources, e.g., from data sources associated with oneor more different computer services. Accordingly, computer system 100may include a user-centric graph server 103 configured to instantiate,maintain, and/or store a collection of user-centric facts as auser-centric graph data structure 300. User-centric graph structure 300represents a plurality of user-centric facts in the form of nodes andedges in a graphical data structure. Structural characteristics ofuser-centric graph structure 300 may indicate facts and/or possiblerelationships between facts. User-centric graph structure 300 mayrepresent each fact as a subject-predicate-object triple including asubject graph node representing a subject of a fact (e.g., an entity,event, or any other noun), an object graph node representing an objectof a fact, and an edge representing a predicate that describes arelationship between the subject and the object. Furthermore,user-centric graph structure may represent relationships between facts,e.g., two or more facts may share a common subject and/or object.Relationships between subjects, objects, and facts in the graph may beencoded as nodes, edges, and/or paths in the graph which may representrelationships indicating a common entity, temporality, and/or topic offacts, as well as any other suitable relationships. The user-centricgraph structure may be useable to derive inferences from one or morefacts, e.g., by traversing the graph along paths formed by edges of theuser-centric facts.

The user-centric graph may be utilized by any computer service, e.g., tostore user-centric facts arising from user interaction with the computerservice, and/or to query the user-centric graph to recognize one or moreuser-centric facts related to interaction with that computer service orany other computer service(s). Computer services which utilize theuser-centric graph may be 1^(st) party computer services authored and/oradministered by an organization or entity which administers theuser-centric graph, or 3^(rd) party services authored and/oradministered by a different organization or entity. Computer servicesmay utilize the user-centric graph via one or more APIs of theuser-centric graph (e.g., an update API and a query API), wherein eachAPI is useable by a plurality of different computer services including1^(st) party computer services and 3^(rd) party services.

A user may interact with a computer service using more than onedifferent device, e.g., via user interfaces for a mobile device, adesktop computer, an intelligent-assistant speaker device, etc. Relevantuser-centric facts may be aggregated from computer services at eachdifferent device with which a user interacts. For example, a user mayinteract with a particular computer service (e.g., email program) atmore than one different device (e.g., via a mobile phone with a mobilephone email application, and via a personal computer device via aweb-browser based email client). Accordingly, the various computerservices may use and/or contribute to the user-centric graph 300,irrespective of which specific device a user may use to interact withthe computer services. For example, although browser 110 is shownassociated with a client computer 102 in FIG. 1A, it will be understoodthat browser 110 may be instantiated on any suitable device, e.g., apersonal computer, a mobile phone, an intelligent assistant device, etc.Accordingly, browser 110 may present assistive information based onuser-centric facts, irrespective of which device instantiated browser110. Because any suitable computer service, running on any suitabledevice, may use user-centric graph 300 to derive assistive information,the presentation of assistive information is agnostic to the particulardevice instantiating browser 110 and/or any other computer service(s).Accordingly, the user may be presented with assistive information in aconsistent fashion, across different devices running browser 110 or anyother browser program.

The user-centric graph server 103 and/or client computer 102 mayconnect, via network 104, to one or more user-centric data stores, e.g.,user-centric data 105A, 105B, and optionally any suitable number offurther user-centric data stores through user-centric data 105N. Eachuser-centric data store may store any suitable data which may bepertinent to the user, e.g., data arising from interaction with one ormore computer services. For example, each user-centric data store may bean application-specific data store which stores data pertaining to anapplication such as user data, configuration settings, and the like.

The user-centric graph data structure 300 includes a plurality ofdifferent constituent graph structures 302, e.g., constituent graph302A, constituent graph 302B, etc, through constituent graph 302N Eachconstituent graph structure may be an application-specific constituentgraph structure associated with a different computer service, e.g.,graph structure 302A associated with user-centric data from a firstcomputer service stored in user-centric data 105A. For example,application-specific constituent graph structure A may be associatedwith a schedule planning program, while application-specific constituentgraph structure B may be associated with an email program. The multipleconstituent graphs may be treated as a single graph by regarding edgesbetween constituent graphs as edges in a single larger graph 300. Tosimplify explanation, the depicted graph only includes twelve nodes. Inactual implementations, a user-centric graph will include many morenodes (e.g., hundreds, thousands, millions, or more) spread between manymore constituent graphs.

User-centric graph server 103 may further connect, via network 104, toany other suitable computer devices and/or computer networks, e.g.,computers on the Internet such as web search providers, web site hosts,etc. In some examples, user-centric graph 300 may be configured toinclude one or more external facts derived from such other computerdevices in addition to user-centric facts pertaining specifically to auser. For example, user-centric graph 300 may include one or more factsderived from a web search result and/or web site. When user-centricgraph 300 includes external facts, the external facts may be related touser-centric facts in the user-centric graph in any suitable manner,e.g., user-centric graph may include one or more nodes and/or edgeswhich represent global data derived from the Internet, which areconnected via paths in the graph to one or more nodes and/or edgesrepresenting user-centric data arising from user interaction with acomputer service. In some examples, connections between user-centricfacts and external facts can create additional/enhanced user-centricfacts.

Client computer 102 may communicate, via network 104, with user-centricgraph server 103 to leverage information represented in the user-centricgraph, in order to assist a user. For example, such assistance mayinclude answering questions for and/or about the user by retrievinguser-centric and/or external facts, looking up historical datapertaining to interactions between the user and one or more computerservices, and/or inferring new facts to add to the user-centric graph.

As shown in FIG. 1A, browser 110 may utilize personal information of theuser from one or more emails and/or calendar/scheduling items, as shownin information pane 121. Information pane 121 can show informationrelated to any one or more suitable contexts (e.g., related to tasks theuser may wish to complete). A context may be based on one or moreuser-centric facts in the user-centric graph. For example, a context maybe based on a cluster of user-centric facts that are mutually connectedby edges within the user-centric graph. Alternately or additionally, acontext may be based on one or more user-centric facts within theuser-centric graph that adhere to a particular constraint, e.g.,matching a particular topic, relating to a particular entity or event,or having a particular temporality (e.g., facts describing eventsoccurring within a particular duration of time, with a particularperiodicity or schedule, or any other suitable temporal relationship).Non-limiting examples of user-centric facts that may be included in thecontext are user-centric facts describing: 1) emails to or from theuser; 2) contacts of the user; 3) calendar data of the user; 4)documents (e.g., including multimedia documents such as photos, videos,or any other content being curated, edited, and/or originated by theuser); 5) web pages; and/or 6) location data associated with the user(e.g., location data measured by sensors of a device belonging to theuser, and/or location data inferred from interaction with a computerservice by the user); and/or 7) application usage data associated withthe user (e.g., login data, usage statistics concerning usage frequencyand duration, etc.; for example, application usage data may includebrowser usage data of 110 including recent page visits, search history,etc.).

A context in the user-centric graph may indicate further user-centricfacts in the user-centric graph 300, e.g., user-centric facts that areconnected via edges/paths in the user-centric graph 300 to one or moreuser-centric facts of the context. For example, a first user-centricfact in the context may be the fact that the user received a first emailfrom an airline booking confirmation email address. A seconduser-centric fact in the context may be the fact that the user receiveda second email from the same airline (e.g., as indicated by the secondemail being from the same or a similar email address as the firstemail). Accordingly, the two user-centric facts may be connected in theuser-centric graph based on both emails being from the same airline. Inaddition to connections in the user-centric graph based on a commonidentifier such as an email address, further examples of connections inthe user-centric graph, based on topical, temporal, and/or otherconstraints will be discussed below.

The user-centric graph 300 may be traversed and/or processed to retrieveone or more user-centric facts related to the context. Traversing and/orprocessing the user-centric graph 300 may be based on structure of theuser-centric graph 300 and/or details of user-centric facts withinand/or related to the context.

For example, the user-centric graph 300 may be traversed by performing arandom walk through the user-centric graph 300, starting with an initialset of one or more user-centric facts in the context and traversingfurther user-centric facts that are connected to those in the initialset, to build up a larger set of potentially relevant user-centricfacts. Randomly walking through the graph may be weighted based onrelationships between the user-centric facts, e.g., details associatedwith subject graph nodes, object graph nodes, and/or edges of theuser-centric facts. For example, user-centric facts may be annotatedwith confidence scores which may be used to weight the random walk.

In addition to performing random walks based on confidence valuesassociated with nodes and/or edges in the user-centric graph, theuser-centric graph may be traversed and/or processed in any othersuitable fashion. Non-limiting examples of processing may include 1)“ranking” user-centric facts in the graph according to a scoringfunction and retrieving relatively higher-ranked user-centric facts(e.g., finding user-centric facts about the most relevant entities withregard to the context), and 2) “slicing” the user-centric graph tofilter user-centric facts that satisfy a particular constraint (e.g.,finding email-related user-centric facts that are connected to thecontext). “Ranking,” “slicing,” and other approaches totraversing/processing the user-centric graph will be described infurther detail below with regard to FIGS. 4A-4D.

The displayed assistive information in pane 121 may be based onuser-centric facts in the context, and/or user-centric facts retrievedfrom user-centric graph 300 based on the context. For example, theassistive information in pane 121 may include a natural languagerepresentation of each of the one or more user-centric facts, and/or anatural-language summary of the one or more user-centric facts. In someexamples, the assistive information may include a user-centric factrelated to completing a task and/or topic related to the context. Insome examples, the assistive information may include a summary ofhistorical activity by the user related to the context.

In addition to being based on user-centric facts, the displayedassistive information may be further be based on external and/or derivedfacts. Non-limiting examples of external and/or derived facts upon whichthe assistive information may be based include: 1) results of a websearch related to the user-centric facts, 2)dictionary/encyclopedia/reference information related to theuser-centric facts, 3) website content related to the user-centricfacts, and/or 4) results of processing the user-centric facts accordingto programmatic, mathematical, statistical, and/or machine learningfunctions (e.g., computing a date or duration based on dates associatedwith user-centric facts in the context, for example, to suggest apossible scheduling of a future task; computing a natural-languagesummary or an answer to a query using a machine learning model; etc).Non-limiting examples of information that may be present within theassistive information includes information based on (e.g., summarized,inferred, and/or otherwise determined from) one or more of 1) emails toor from the user; 2) contacts of the user; 3) calendar data of the user;4) documents; 5) web pages; 6) location data associated with the user;and/or 7) application usage data associated with the user.

Browser 110 may be configured to present information related to one ormore of a plurality of pre-defined contexts, e.g., “travel” and“upcoming tasks.” For example, as shown in FIG. 1A, information pane 121shows assistive information related to a travel itinerary and upcomingtasks from the user's calendar. Alternately or additionally, browser 110may be configured to intelligently recognize one or more contexts ofparticular relevance to a user, based on personal information of theuser. For example, even when browser 110 may not be configured topresent contextual information about a “travel” topic, browser 110 maynonetheless recognize “travel itinerary” as a relevant context of theuser, e.g., based on the user's email and/or calendar data.

By presenting relevant assistive information responsive to thedetermined context (including relevant flight dates from emails to theuser, and an upcoming scheduled task from the user's calendar), thebrowser may help the user to determine a suitable hotel booking datefrom available dates in calendar 114, without requiring the user toseparately find the user's personal email and calendar data.Accordingly, the user may be able to complete a selection in calendar114 and confirm the selection with button 115, without needing toredirect attention to intermediate steps such as opening up an emailprogram and finding a relevant email, and/or opening up a calendarprogram and navigating to relevant dates. In the present example,presenting the flight dates and upcoming tasks may save the user severalsteps. For example, the user may be aware that they have scheduled avideoconference sometime during the trip and may need to plan to be at ahotel with wireless internet during the scheduled time, but may notremember the exact dates of the trip. Accordingly, to plan the hotelbooking, the user may need to figure out the flight dates (e.g., fromrelevant emails), then check the calendar during the relevant dates. Asshown in FIG. 1A, computing system 100 may intelligently determine thatthe user may require the relevant information from emails/calendar andpresent assistive information related to the context in information pane121, thereby allowing the user to avoid the steps of searching email forthe relevant flight booking and then searching a calendar during therelevant duration.

In some examples, a context for triggering assistive information may bedefined by a collection of user-centric facts related to the context,e.g., tasks, emails, contact info, web sites, web searches,file/database searches, historical browser activity, historical activityin other applications, etc. In some examples, the collection ofuser-centric facts may include information content that is semanticallyrelated, e.g., a collection of user-centric facts defining a context fora predefined topic may include user-centric facts that are eachsemantically related to the topic. For example, when a predefined topicis “travel,” the collection of user-centric facts may include emails,web searches, etc., that each include semantic data and/or metadatarelated to travel. Non-limiting examples of semantic data includenatural language content, image content, text, numerical data, etc.Non-limiting examples of metadata include file/web/database locations,publication dates, access times, authorship information,sender/recipient information, etc.

In some examples, a context may be defined by a collection ofuser-centric facts that are related by a temporal relationship, based onthe user-centric facts being associated with related date/timeinformation. A collection of user-centric facts defining a context basedon a temporal relationship may be recognized, even if the collection ofuser-centric facts would not otherwise be recognized as defining acontext based on a task/activity, ongoing project, predefined topic,and/or topic based on recent activity. Non-limiting examples of contextsbased on temporal relationships include: 1) a co-occurring collection ofweb sites, emails, etc., that were accessed by the user simultaneouslyor within a short duration of each other; 2) an activity with anassociated scheduled time (e.g., a due-date for a project or a scheduledappointment), along with one or more other activities that occurred orare scheduled to occur soon before or after the scheduled time; 3) arecurring temporal context, e.g., every evening, every Tuesday at 5:00PM, or any other suitable scheduled and/or periodic occurrence.

For example, when a user is travelling, they may also need toparticipate in a scheduled teleconference. Accordingly, a “travel”context may also include the temporally-related information for theteleconference schedule, since the scheduled date for the teleconferenceoccurs within the travel dates, even if the teleconference schedulewould not otherwise be recognized as being semantically related to thetravel itinerary.

Recognizing a context for triggering assistive information may includecombining multiple different signals, e.g., metadata, semantic content,and temporal relationships, to assess a strength of relationship betweendifferent user-centric facts. If a collection of user-centric facts isrelated in more than one different way (e.g., similar topic, as well asa temporal relationship), it may represent a more significant contextfor the user. In some examples, the presence of multiple signals mayallow the recognition of a context that would otherwise not berecognized.

FIG. 1B shows another example of a browser session in which a user iscurrently editing a document in browser 110 via an in-browser editor, asshown in tab 111′ displaying “DOCUMENT 1.” In the exemplary session, theuser also recently interacted with a web search tool in tab 112′ marked“WEB SEARCH.” The currently displayed tab 111′ includes page content113′ in the form of an editable interface. The editable interface mayallow the user to edit and save displayed content, e.g., using a wordprocessing GUI including a cursor 116 for selecting a position and/orrange in the text. As shown, the “DOCUMENT 1” contents shown in pagecontent 113′ include user-defined text describing an “R&D PROJECT” andincluding various notes about the project logistics, e.g., noting thatAlice, Bob, and Carol are assigned to a project that is due on the date9/7 according to an email from David.

Based on page contents 113′ and recent interactions between the user andone or more other applications, info button 120′ may indicate thatinformation pane 121′ of assistive information is available for acurrent context. The current context may be defined by user-centricfacts relating to any suitable information about the user's interactionwith the browser, for example, user-edited contents in page content113′, previous interactions between the user and the browser such as websearches conducted via tab 112′, other historical web site visits and/orweb searches, and/or recent interactions between the user and otherapplications, such as other document editing applications, emailapplications, and/or scheduling applications. In particular, the currentcontext may be defined by one or more user-centric facts related to the“R&D PROJECT.” As shown in information pane 121′, assistive informationrelated to the “R&D PROJECT” is displayed based on such context, e.g.,so as to potentially assist the user with completing one or more tasksrelated to the R&D project.

The assistive information shown in information pane 121′ may include anysuitable information which may be relevant to the context. For example,information pane 121′ includes: 1) a listing of related documents thatthe user may wish to view and/or edit to complete the R&D project; 2) arelated web site describing a competitor's approach; 3) related websearches pertaining to the competitor's approach; 4) related tasks thatthe user may wish to complete in order to complete the R&D project; 5)related emails pertaining to the project; and 6) people who the user maywish to contact pertaining to the project.

For example, related emails may be determined by one or moreuser-centric facts defining the context based on user interaction withan email application. Related documents may be determined by one or moreuser-centric facts in the context that were based on user interactionwith the browser (e.g., with a document editor displayable in thebrowser such as in page content 113′) and/or with any other computerservices (e.g., with a dedicated word processor or slide presentationprogram). Similarly, the listed people may be determined based on emailsto and/or from the user, collaborators who also edited a relateddocument, and/or people listed in a contact list of the user. Byidentifying a context and relevant documents, tasks, web sites, websearches, tasks, emails, and people, browser 110 may be able tofacilitate easier completion of the R&D project, since the user may beable to access relevant information from information pane 121′ insteadof explicitly consulting various sources of information (e.g.,documents, contact lists, emails, etc.). Accordingly, the user can findand use relevant information from multiple different sources, withoutdiverting attention from working on “DOCUMENT 1” in tab 111′.

Similarly, a collection of user-centric facts defining a context for atopic based on recent activity may include user-centric facts that eachrelate to one or more common topics. Determining that user-centric factsbelong in a collection defining a particular context may also be basedon any other suitable information related to the user-centric facts,e.g., metadata and semantic content of the user-centric facts.

FIG. 2 shows an exemplary method 200 for providing contextualinformation in a browser to assist a user. Method 200 includes, at 210,maintaining a user-centric graph including a plurality of user-centricfacts associated with a user. The user-centric facts may be derived frominteraction, by the user, with one or more different computer services.

Method 200 may be performed by any suitable combination of devices ofFIG. 1A. For example, method 200 may be performed by executing codestored at one or more storage devices associated with any of the devicesof FIG. 1A, on one or more processors of any of the devices of FIG. 1A.Furthermore, the processors and/or storage devices of devices shown inFIG. 1A may be operated in any suitable combination to instantiateuser-centric graph 300 and/or browser 110. For example, browser 110 mayinclude a graphical user interface visually presented at a displaydevice of client computer 102, and processing functionality provided byany combination of client computer 102 and other devices of FIG. 1A.Operating browser 110 at client computer 102 may generally include 1)gathering information related to user interaction with browser 110 thatis useable to define and/or infer a context in user-centric graph 300,2) sending such information to one or more other devices of FIG. 1A(e.g., to user-centric graph server 103), 3) receiving responses fromthe one or more other devices, wherein the responses include assistiveinformation based on the context, and 4) visually presenting theresponses in the GUI for browser 110.

At 230, method 200 further includes recognizing a context of interactionwith a browser application by the user. Optionally, at 232, the contextis defined by at least one or more contextualizing user-centric factsderived from interaction, by the user, with one or more differentcomputer services. Optionally, at 234, recognition of the context istriggered by recognition of a triggering fact. The triggering fact maybe any user-centric fact, or any other fact (e.g., a fact derived from aweb site being visited by the user). The context may be based on thetriggering fact in any suitable manner. For example, the context may bedefined by a single user-centric fact, namely, the triggering fact.Alternately or additionally, the context may be defined by one or morefacts that are related to the triggering fact in any suitable manner(e.g., via structural relationships of the graph, commontopics/tasks/entities/events, etc., as described above). For example,the context may be defined by a temporal relationship between thetriggering fact and one or more other user-centric facts in the graph(e.g., based on a temporal relationship between a date associated with auser interaction defining the triggering fact and/or other user-centricfacts, a due-date associated with a user-centric fact, etc.).

The triggering fact may be based on one or more previous interactions(e.g., recent and/or historical interactions), by the user, with thebrowser or any other computer service(s). For example, user centricgraph server 103 may be configured to receive, from browser 110, acomputer-readable description of one or more previous interactions, bythe user, with the browser. Accordingly, the triggering fact may includesuch computer-readable description of the previous interaction(s).Non-limiting examples of interactions between the user and the browserinclude: 1) metadata and/or content corresponding to a recent filelocation and/or web site visited by the user; 2) a recent file and/orweb search query submitted by the user; 3) content input by the user ina web form, editable document, or via any other input mechanism providedby the browser; 4) closing one or more tabs, windows, or other interfaceelements of the browser; 5) clicking on an information button to requestinformation (e.g., clicking on button 120 of browser 110); 6) viewinglocation/search history in the browser; 7) adding a new bookmarked (e.g.“favorite”) page and/or viewing a listing of bookmarked pages; 8)interacting with a web site or any other computer service (e.g., placingan order to purchase a product, booking a reservation, sending and/orreading an email via an email client accessible via the browser, etc.);and/or 9) a previous interaction, by the user, with a computer serviceother than the browser (e.g., an email or calendar program).

In some examples, the browser may continually recognize potentialtriggering facts arising from user interaction with the browser, inorder to determine whether, for each candidate triggering fact,assistive information should be presented based on a context related tothe triggering fact. For example, recognizing the context may includeassessing, for a candidate triggering fact, a signal strength estimatinga relevance of the triggering fact and/or context(s) related to thetriggering fact. The signal strength may be based on the triggering factand/or a context related to the triggering fact. In some examples,assessing a signal strength may be based on a confidence valueassociated with the triggering fact and/or other user-centric facts in acontext relating to the triggering fact.

In some examples, assessing a signal strength may be based on one ormore graph-theoretic characteristics or other structural properties ofthe user-centric graph, e.g., an amount of clustering or connectivity ofthe context relating to the triggering fact. Non-limiting examples ofgraph-theoretic characteristics that may be suitable for assessing asignal strength include 1) connectivity, e.g., number of connectedcomponents and/or k-connected components for any suitable value of ksuch that k vertices need to be removed to disconnect the k-connectedcomponents); 2) path distance; 3) presence and/or absence of cycles; 4)graph-theoretic strength and/or toughness; and/or any other suitablegraph-theoretic properties. In some examples, browser 110 may recognizea triggering fact and display assistive information only for candidatetriggering facts and contexts for which a relatively high signalstrength is assessed. For example, recognizing the triggering fact mayinclude assessing, for a candidate fact, a signal strength scoreindicating a strength of relationship between the candidate fact anduser-centric facts in the user-centric graph, and recognizing thetriggering fact responsive to the signal strength score exceeding athreshold. Alternately or additionally, recognizing the triggering factmay include recognizing, as the triggering fact, a candidate triggeringfact having a highest signal strength among a plurality of triggeringfacts occurring in user interaction with the browser.

In an example, a context relating to a predefined topic may be definedby at least one node in the user-centric graph which has outgoing edgesconnected to one or more other nodes based on the one or more othernodes each pertaining to the predefined topic. In an example, a contextdefined by a topic may be recognized based on structural properties ofone or more user-centric facts in the user-centric graph, e.g., based ongraph theoretic characteristics such as the user-centric facts forming adense cluster, or any other suitable graph-theoretic characteristics asdescribed above with regard to assessing signal strength for atriggering fact.

At 240, method 200 further includes evaluating the context to determineassistive information pertaining to the context. Optionally, at 242, theassistive information is based at least one or more user-centric factsderived from interaction, by the user, with one or more differentcomputer services. Optionally, at 244, the assistive information isbased at least on a relationship, in a user-centric graph, between atriggering fact and one or more other user-centric facts in theuser-centric graph. In some examples, after recognizing a triggeringfact, user-centric graph 300 may be updated to include the triggeringfact. In some examples, responsive to recognizing a triggering fact,user-centric graph 300 may be updated to include one or more additionaluser-centric facts based on aggregating, summarizing, and/or otherwiseanalyzing other user-centric facts in user-centric graph 300. Forexample, if the user is frequently interacting with travel-relatedwebsites as shown in FIG. 1A, user-centric graph 300 may be updated toinclude an additional user-centric fact summarizing the user'sinteractions by stating that “the user frequently researches travel.”This new user-centric fact may be related, via paths in the user-centricgraph 300, to other user-centric facts relating to travel. The newuser-centric fact may define a context which may be used to returnassistive information to the user, e.g., a “travel” topic context withrelated assistive information including previous travel-relatedinteractions by the user, as well as other general information relatedto travel (e.g., travel web sites, travel-related web searches, etc.).

At 250, method 200 further includes displaying the assistive informationto the user. The assistive information can be displayed in any suitablefashion. For example, as shown in FIG. 1A-1B, the assistive informationmay be displayed as a listing of facts grouped into differentcategories, e.g., “Travel Itinerary,” “Upcoming Tasks,” “RelatedDocuments,” “Websites,” “Web Searches,” “Tasks,” “Emails,” and “People”categories as shown in FIGS. 1A-1B. By recognizing the context of theuser and displaying the assistive information related to that context,the browser may facilitate easier completion of tasks pertaining to thecontext. Accordingly, method 200 may improve the efficiency and/or easeof use of one or more computers.

FIGS. 3A-3B show details of an exemplary graph data structure forrepresenting a collection of user-centric facts as a user-centric graph.FIG. 3A shows exemplary graph data structure 300 in more detail in theform of a computer-readable data format 300′ for the graph datastructure 300. Graph data structure 300 represents a collection ofuser-centric facts in data format 300′ or any other suitable dataformat. The user-centric facts may be associated withapplication-specific data distributed across a plurality of differentcomputer services. The graph data structure 300 allows centralizedqueries and is suitable for implementing a user-centric graph.

Graph data structure 300 may be distributed across one or moreapplication-specific constituent graph structures (e.g., correspondingto different computer services), such as constituent graph structure 302indicated in a first region surrounded by a broken line, A, constituentgraph structure 302B indicated in a second region surrounded by adifferent broken line, and other constituent graph structures not shownin FIG. 3A.

Each constituent graph structure includes a plurality of user-centricfacts 304, e.g., user-centric facts F_(A.1), F_(A.2), etc. stored inconstituent graph A, and user-centric facts F_(B.1), F_(B.1), etc.stored in constituent graph B. A user-centric fact includes a subjectgraph node 306, an object graph node 308, and an edge 310 connecting thesubject graph node to the object graph node.

Subject graph nodes and object graph nodes may be generically referredto as nodes. Nodes may represent any noun, where “noun” is used to referto any entity, event, or concept, or any suitable application-specificinformation (e.g., details of a previous action performed by the userusing the computer service). Similarly, “subject noun” and “object noun”are used herein to refer to nouns which are represented by a subjectgraph node or by an object graph node respectively. Representing thecollection of user-centric facts as a graph data structure mayfacilitate manipulating and traversing the graph data structure (e.g.,to respond to a query).

It is instructive to visualize the collection of user-centric facts in agraphical format as shown in graph data structure 300. Nodes 352 aredepicted as filled circles and edges 354 are depicted as arrows. Filledcircles with an outgoing edge (where the arrow points away from thefilled circle) depict subject graph nodes, while filled circles with anincoming edge (where the arrow points towards the filled circle) depictobject graph nodes. The multiple constituent graphs may be treated as asingle graph by regarding edges between constituent graphs as edges inthe larger graph. Accordingly, FIG. 3B shows a single combined graph 300including a plurality of constituent graph structures. To simplifyexplanation, example graph 300 only includes two constituent graphs,with twelve nodes in total. In actual implementations, a user centricgraph will include many more nodes (e.g., hundreds, thousands, millions,or more) spread between many more constituent graphs.

An exemplary user-centric fact F_(A.1) of constituent graph structureincludes subject graph node S_(A.1), edge E_(A.1), and object graph nodeO_(A.1). For example, subject graph node S_(A.1) may represent theuser's employer and object graph node O_(A.1) may represent a taskassigned to the user by her employer. The edge E_(A.1) may describe anysuitable relationship between subject graph node S_(A.1) and objectgraph node O_(A.1). In the above example, edge E_(A.1) may represent an“assigned new task” relationship. The subject-edge-object triple ofuser-centric fact F_(A.1) collectively represents the fact that theuser's employer assigned her a new task.

A subject graph node of a first fact may represent the same noun as anobject graph node of a second, different fact. Accordingly, said subjectgraph node and object graph node may be represented as a single node,e.g., a node 322 with an incoming arrow and an outgoing arrow asdepicted in FIG. 3B. By recognizing that certain object graph nodes andsubject graph nodes represent the same noun, the graph data structuremay be able to represent user-centric facts as complex relationshipsamong a plurality of different nouns, which may be visualized as pathson graph 300′. For example, when a particular node is the object graphnode of a first fact and the subject graph node of a second, differentfact, it may be possible to derive inferences from the combination ofthe two facts, analogous to a logical syllogism.

A subject graph node of a first user-centric fact may represent the samenoun as a subject graph node of a second, different user-centric fact.When two different subject graph nodes represent the same noun, thegraph data structure may recognize the two subject graph nodes as asingle node, e.g., a node 324 with two outgoing arrows in FIG. 3B.Although the two subject graph nodes may be represented as a singlenode, each outgoing edge of the node is distinct from each otheroutgoing edge. These outgoing edges may point to the same object node orto different object nodes. In either case, the subject node may beinvolved in two different user-centric facts.

Similarly, an object graph node of a first user-centric fact mayrepresent the same noun as an object graph node of a second, differentuser-centric fact. In other words, the same noun may be the object of aplurality of different user-centric facts, having different subjectgraph nodes and possibly having edges representing differentrelationship types, e.g., as with node 326 with two incoming arrows inFIG. 3B. Accordingly, the graph data structure may recognize the twoobject graph nodes as a single node, which is depicted in the sameposition in the graph 300.

A particular pairing of a subject noun and an object noun may beinvolved in two or more different user-centric facts. For example, thesubject noun and object noun may be represented by the pair of nodesindicated at 328, which are connected by two distinct edges, e.g.,representing distinct relationships between the subject and object graphnodes. In an example, the subject graph node corresponds to a useraccount (e.g., identified by an email address). In the same example, theobject graph node may correspond to a second, different user account. Inthe example, a first edge may represent a “sent email to” relationship,while a second edge represents a different “scheduled meeting with”relationship. Accordingly, the graph data structure includes two or moreuser-centric facts having the same subject and object nouns.

In other examples, two nouns may be involved in two differentuser-centric facts, but with the role between subject and objectswapped. In other words, a first noun is a subject noun of a first factand a second noun is an object noun of the first fact, while the firstnoun is an object noun of a second fact and the second noun is a subjectnoun of the second fact. For example, a pair of nouns representing“Alice” and “Bob” might be involved in a first fact saying that “Alicescheduled a meeting with Bob” while also being involved in a second factsaying that “Bob scheduled a meeting with Alice.” In addition toswapping a role of subject and object, the two facts using the two nounsmay have different types of edges, e.g., “Alice” and “Bob” mightadditionally be involved in a third fact saying that “Bob sent an emailto Alice.” More generally, nouns may be connected via edges to representrelationships in any suitable fashion, e.g., with any number of edgesbetween two nodes, going between the first and second node in anycombination of forward and/or backward direction.

In still further examples, the graph data structure may recognize a nounfrom one constituent graph and a noun from a different constituent graphas a single node. Such recognition that two or more different nodescorrespond to the same noun may be referred to herein as a “nodecross-reference” between the two nodes. A subject graph node or objectgraph node representing a particular noun may store nodecross-references to other nodes of the same constituent graph structureor any other constituent graph structure.

Similarly, a user-centric fact in a first constituent graph structuremay define a subject graph node in the first constituent graphstructure, along with an edge pointing to an object graph node in asecond, different constituent graph structure (herein referred to as an“cross-reference edge”). The user-centric graph may represent theconnection between the constituent graphs in any suitable manner, forexample, by storing a constituent graph identifier in constituent graphA, the identifier indicating a connection to an object graph node inconstituent graph B, and an object graph node identifier indicating theparticular object graph node in constituent graph B.

Node cross-references and cross-reference edges connect the plurality ofconstituent graph structures. For example, node cross-references andcross-reference edges may be traversed in the same way as edges,allowing a traversal of the graph data structure to traverse a pathspanning across multiple constituent graph structures. In other words,graph data structure 300 and/or data format 300′ facilitate a holisticartificial intelligence knowledge base that includes facts fromdifferent computer services and/or facts spanning across differentcomputer services. Node cross-references and cross-reference edges maybe collectively referred to herein as cross-references. Similarly, whena node is involved in cross-references within a plurality of constituentgraph structures, the node may be referred to as cross-referenced acrossthe constituent graph structures.

Furthermore, in addition to connecting two different constituent graphstructures via cross-references, a constituent graph structure mayinclude one or more user-centric facts involving a subject graph node inthe constituent graph structure and an object graph node in an externalknowledge base (e.g., based on the Internet, a social network, or anetworked, enterprise computer service). As with cross-reference edges,an outgoing edge connected to the subject graph node may indicate aconnection to an external object graph node in any suitable manner,e.g., by storing a pair of identifiers indicating the external graph andthe object graph node within the external graph. In some cases, theexternal graph may not store any user-centric data, e.g., when theexternal graph is a global knowledge base derived from publishedknowledge on the Internet.

By including cross-references between constituent graph structures,facts about a particular noun (e.g., event or entity) may be distributedacross the plurality of constituent graph structures, while stillsupporting centralized reasoning about the relationship betweenuser-centric facts in different constituent graph structures and inexternal databases (e.g., by traversing the plurality of constituentgraph structures via the cross-references and cross-reference edges).

Each user-centric fact may be stored in a predictable, shared dataformat, which stores user-centric facts including application-specificfacts associated with a computer service without requiring a change tothe format of application-specific data the computer service. Such apredictable, shared data format is herein referred to as an“application-agnostic data format.” The application-agnostic data formatmay store information needed to query the user-centric graph, whileavoiding the redundant storage of application-specific data. The graphdata structure may be implemented with a complementary applicationprogramming interface (API) allowing read and write access to the graphdata structure. The API may constrain access to the graph data structureso as to ensure that all data written to the graph data structure is inthe application-agnostic data format. At the same time, the API mayprovide a mechanism that any computer service may use to add newuser-centric facts to the graph data structure in theapplication-agnostic data format, thereby ensuring that all data storedwithin the graph data structure is predictably useable by other computerservices using the API. In addition to providing read/write access touser-centric facts stored in the graph data structure, the API mayprovide data processing operations that include both reads and writes,e.g., query operations and caching of query results.

A graph data structure including user-centric facts related to aplurality of different computer services, such as graph data structure300, may be variously implemented without departing from the spirit ofthis disclosure. For example, the graph data structure may be stored asa collection of node records, each of which optionally indicates one ormore outgoing edges to other nodes. Alternately or additionally, thegraph data structure may be stored as a pointer-based data structure,list data structure, matrix data structure, or as any other suitablerepresentation of graphical data.

In an example, a user-centric fact stored in the graph data structure isan application-specific fact. In addition to the application-specificfact, the user-centric fact may include one or more enrichments, wherean enrichment is an additional datum that includes anapplication-agnostic fact associated with the application-specific fact.For example, an enrichment may include a browser-specific enrichmentbased on details of user interaction with a browser program (e.g.,browser 110). The browser specific enrichment may include informationdescribing a single interaction between the user and the browser, awhole session of user/browser interactions, and/or any suitable set ofhistorical user/browser interactions. In the example shown in FIG. 1A, abrowser-specific enrichment may include a computer-readable descriptionof the user's visit to a travel booking page. Alternately oradditionally, the browser-specific enrichment may include an analysis offurther interactions between the user and the browser. For example, theenrichment may include a description that the user frequently visitedthe travel booking page within the last week, along with a descriptionof the frequency and specific times the user visited the travel bookingpage. Triggering facts, user-centric facts defining contexts, and otheruser-centric facts in the user-centric graph may include any suitablebrowser-specific or other application-specific enrichments. Accordingly,assistive information presented by the browser may be based on suchenrichments.

The application-agnostic data format for user-centric facts permitsefficient storage of application-specific facts, in addition to theapplication-agnostic representation of the connections in the graph datastructure. For example, a subject graph node may represent a subject ofan application-specific fact, e.g., the fact that a new meeting wasscheduled. Accordingly, the graph data structure may store for theapplication-specific fact a facet pointer that indicates auxiliaryapplication-specific data associated with the application-specific fact.In an example, auxiliary application-specific data is indicated in theform of a facet pointer. A facet point is an identifier (e.g., a numericidentifier) indicating a storage location of the auxiliaryapplication-specific data associated with the user-centric graph, e.g.,a storage location of a calendar entry representing details of themeeting. A facet pointer may be associated with a particular type ofdata based on its inclusion in an application-specific graph structure,e.g., constituent graph structure A may be associated with scheduleplanning software and may accordingly include calendar data as opposedto other types of application-specific data. In other examples, a facetpointer may include additional identifying information specifying a typeof the auxiliary application-specific data, so that facet pointers maybe used to represent different kinds of auxiliary application-specificdata (e.g., multiple file types useable by a word processingapplication). The graph data structure may be used to find auxiliaryapplication-specific data via the facet pointers, while avoidingredundantly storing the auxiliary application-specific data within thegraph data structure.

In addition to the facet pointer stored in a subject graph node, anapplication-specific fact is further represented by the edges connectingthe subject graph node to one or more object graph nodes. Although asingle subject graph node may be included in more than one user-centricfact, the graph data structure nevertheless efficiently stores only asingle node record for the subject graph node. This node record includesthe list of all of the node's outgoing edges, which may reduce a storagespace requirement compared to storing a copy of the subject graph nodefor each user-centric fact in which it occurs. The list of outgoingedges may be empty for some nodes, e.g., for a noun that is only anobject graph node. The list of outgoing edges of a subject graph nodeincludes one or more edge records defining one or more edges. A graphnode may have any number of outgoing edges, to suitably representrelationships to other nodes of the graph data structure, e.g., zero,one, or three or more edges.

In addition to representing an application-specific fact via the facetpointer and the list of outgoing edges, a subject graph node mayrepresent the one or more enrichments of the application-specific fact.In an example, the one or more enrichments include a node confidencevalue, which may indicate a relevance of a graph node to the user. Forexample, the confidence value may be determined by a machine learningmodel trained to recognize relevance to the user, by learning todistinguish labelled samples of relevant data from labelled samples ofirrelevant data. For example, training the machine learning model mayinclude supervised training with user-labelled samples (e.g., derivedfrom direct user feedback during application with an application),and/or unsupervised training. The one or more enrichments may furtherinclude an edge confidence value associated with each edge. As with nodeconfidence values, an edge confidence value may indicate a relevance ofa particular edge to the user. Different edges between a pair of nodesmay have different confidence values. For example, an edge indicating a“scheduled meeting” relationship may have a lower edge confidence than adifferent edge indicating a “made commitment,” e.g., if the scheduledmeeting is believed to be more relevant to the user than the commitment.

In addition to a node confidence value and edge confidence value, theone or more enrichments of the application-specific fact may includeother application-agnostic and/or application-specific data. Forexample, a node may include access metadata indicating informationassociated with accessing the node in the user-centric graph, andassociated application-specific data (e.g., data indicated by a facetpointer). Access metadata may include a timestamp indicating a time anddate of a most recent access, a delta value indicating a change causedby a most recent access, or any other suitable metadata.

The graph data structure may additionally store, for a user-centricfact, one or more tags defining auxiliary data associated with theuser-centric fact, including any suitable auxiliary data associated witha node/edge. For example, when a graph node represents a person (e.g.,associated with a contact book entry), the tags may include a nicknameof the person and an alternate email address of the person.

User-centric facts and nodes/edges of the user-centric facts may beenriched with additional semantics stored in the tags. For example, tagsmay be used to store one or more enrichments of a user-centric fact. Atag stored within a node record may be associated with a user-centricfact in which the node record represents a subject graph node or inwhich the node record represents an object graph node. Alternately oradditionally, a tag stored within a node record may be associated withthe node record itself (e.g., associated with a subject graph noderepresented by the node record and/or with an object graph noderepresented by the node record), or with one or more edges connected tothe node record. In other examples, tags may be used to store metadataof a user-centric fact (e.g., instead of or in addition to the accessmetadata of the user-centric fact). For example, when a user-centricfact is associated with a timestamp, the timestamp may optionally bestored among the tags of the user-centric fact.

In some examples, the one or more tags are searchable tags and the graphdata structure is configured to allow searching for a user-centric factby searching among the searchable tags (e.g., searching for a tagstoring a search string, or searching for a tag storing a particulartype of data).

Graph data structures may be represented as a plurality ofapplication-specific constituent graph structures, wherein nodes of theconstituent graph structures are cross-referenced across constituentgraph structures (e.g., by cross-reference edges and nodecross-references, as described above).

The user-centric graph is application-agnostic, in that it may be usedto track facts from two or more potentially unrelated computer services,which may have different native data formats. The user-centric graph maystore application-specific facts of any computer service by storing afacet pointer, enabling user-centric facts to includeapplication-specific facts even when the application-specific facts maybe stored in an application-specific format. Moreover, the user-centricgraph enables the representation of user-centric facts that are definedin a context of two or more computer services, by storingcross-references in the form of domain identifiers (e.g., object domainidentifiers in lists of outgoing edges of each subject graph node, orreference domain identifiers in a representation of a nodecross-reference) and node identifiers (e.g., object graph nodeidentifiers and reference node identifiers). Furthermore, theuser-centric graph is user-centric, as a different user-centric graphmay be defined for each user. The user-centric graph may be suitable tostore user-centric facts in contexts where each of a plurality ofdifferent users interact with a shared computer service (e.g., a webbrowser). Because the user-centric graph stores application-specificfacts via the facet pointer and represents relationships to facts inother data structures via cross-references, it may be able to storeuser-centric facts concerning a user in a user-centric graph datastructure particular to the user, without requiring write access toapplication-specific data of the shared computer service.

When adding a new user-centric fact based on an application-specificfact, the graph data structure optionally may store anapplication-agnostic enrichment, e.g., an application-agnostic factassociated with the application-specific fact. A user-centric fact maybe partially defined by an enrichment received from theapplication-specific data provider. Alternately or additionally, theuser-centric fact as provided by the application-specific data providermay be preprocessed to include one or more enrichments via an enrichmentpipeline including one or more enrichment adapters. When the graph datastructure stores tags associated with a user-centric fact (e.g., tagsstored in a node record), the one or more enrichments may be includedamong the tags.

In an example, an enrichment adapter includes a machine learning modelconfigured to receive an application-specific fact, to recognize arelevance of the application-specific fact to a user, and to output aconfidence value numerically indicating the relevance. The machinelearning model may be any suitable model, e.g., a statistical model or aneural network. The machine learning model may be trained, e.g., basedon user feedback. For example, when the machine learning model is aneural network, output of the neural network may be assessed via anobjective function indicating a level of error of a predicted relevanceoutput by the neural network, as compared to an actual relevanceindicated in user feedback. The gradient of the objective function maybe computed in terms of the derivative of each function in layers of theneural network using backpropagation. Accordingly, weights of the neuralnetwork may be adjusted based on the gradient (e.g., via gradientdescent) to minimize a level of error indicated by the objectivefunction. In some examples, a machine learning model may be trained fora particular user based on direct feedback provided by the user whileinteracting with a computer service, e.g., indicating relevance ofsearch results in a search application. Accordingly, the trained machinelearning model may be able to estimate relevance to the user. In someexamples, the machine learning model may be trained based on indirectfeedback from the user, e.g., by estimating a similarity of relevantcontent to other content that the user indicated to be relevant in thepast.

In another example, an enrichment adapter includes a natural languageprogram for recognizing natural language features of anapplication-specific fact. For example, the natural language program maydetermine a subject graph node and/or object graph node for theapplication-specific fact, by recognizing a natural language feature asbeing associated with an existing subject and/or object graph node. Insome examples, the natural language program may determine a relationshiptype for an edge for the application-specific fact. In some examples,the natural language program may determine one or more tags of a subjectgraph node and/or object graph node for the application-specific fact.The natural language program may be configured to recognize featuresincluding: 1) named entities (e.g., people, organizations, and/orobjects), 2) intents (e.g., a sentiment or goal associated with anatural language feature), 3) events and tasks (e.g., a task the userintends to do at a later time), 4) topics (e.g., a topic that auser-centric fact contains or represents), 5) locations (e.g., ageographic location referred to by a user-centric fact, or a place wherea user-centric fact was generated), and/or 6) dates and times (e.g., atimestamp indicating a past event or a future scheduled event associatedwith a user-centric fact).

The enrichments associated with user-centric facts may provide enrichedsemantics of the user-centric facts (e.g., additional meaningfulinformation, beyond information provided by the connection structureformed by an edge between a subject graph node and object graph node ofthe user-centric fact). The graph data structure may recognize andinclude additional user-centric facts that may be derived from theenriched semantics (e.g., based on one or more enrichments added in theenrichment pipeline). Accordingly, adding a user-centric fact includingone or more enrichments may further include recognizing an additionaluser-centric fact based on the one or more enrichments, and adding theadditional user-centric fact to the graph data structure in theapplication-agnostic data format.

Recognizing the additional user-centric fact based on the one or moreenrichments may include recognizing that an enrichment of the one ormore enrichments corresponds to another user-centric fact alreadyincluded in the graph data structure (e.g., because the enrichment isassociated with a subject noun or object noun of the other user-centricfact). Alternately or additionally, recognizing the additionaluser-centric fact based on the one or more enrichments may includerecognizing a first enrichment of the one or more enrichments that isassociated with a subject noun not yet involved in any user-centricfacts, recognizing that a second enrichment of the one or moreenrichments is associated with an object noun, recognizing arelationship between the subject noun and the object noun, and adding anew user-centric fact involving the object noun and subject noun to thegraph data structure. In some examples, recognizing the additionaluser-centric fact based on the one or more enrichments includesrecognizing any suitable relationship among the one or more enrichments,and adding a user-centric fact representing the recognized relationship.

In an example, a first node and a second node each include an enrichmentspecifying a recognized named entity, wherein both enrichments specifythe same named entity. Accordingly, the graph data structure may storean edge connecting the first node to the second node, and therelationship type of the edge may indicate that the two nodes wereinferred to be associated with the same entity. Alternately oradditionally, the graph data structure may store a node cross-referencein each node, indicating that the other node is associated with the samenamed entity. Alternately, the graph data structure may modify the firstnode to include data of the second node and delete the second node,thereby avoiding the redundant storage of data of the second node bycollapsing the representation to include a single node instead of twonodes.

In another example, a first node includes an enrichment specifying anamed entity, and an edge may be added connecting the first node to asecond node that represents the same named entity. In another example,an edge may be added between a first node and a second node that havethe same associated topic. In another example, an edge may be addedbetween a first node and a second node that have the same associatedtime and/or location. For example, an edge may be added between a firstnode that refers to a specific calendar date (e.g., in a tag) and asecond node that was created on the specific calendar date (e.g., asindicated by access metadata). In another example, an edge may be addedbetween two nodes that were created at the same location or that referto the same location.

There is no limit to the number of different computer services that maycontribute to the knowledge base. Additionally, there is no requirementthat the different computer services be related to each other or to theuser-centric graph in any particular way (e.g., the different computerservices and the user-centric graph may be mutually unrelated andprovided by different computer service providers). Accordingly, theuser-centric knowledge AI base may include user-centric facts derivedfrom a plurality of different computer services, thereby including moreuser-centric facts from more different contexts. Furthermore, thecross-references between application-specific constituent graphstructures enable user-centric facts to express relationships betweenaspects of the different computer services, which may further improveutility compared to maintaining a plurality of different, separateknowledge bases without cross-references.

The graph data structure represents structured relationships between theuser-centric facts (e.g., two facts having the same subject noun may berepresented by a single node record within a constituent graph andcross-referenced between constituent graphs). As such, it may be moreefficient to traverse the graph data structure to find user-centricfacts satisfying the query, than it would be to exhaustively search thecollection of user-centric facts. For example, if a user has frequentlyinteracted with a particular other person, the frequent interaction mayindicate that the other person is likely relevant to the user.Accordingly, there may be more user-centric facts having that person assubject or object, and while traversing edges of the graph datastructure, encountering a node representing the other person may be morelikely because of the many edges leading to and from the noderepresenting the other person.

Traversing the graph data structure may include a “random walk” alongedges of the graph data structure. The random walk may start at acurrent user context, used herein to refer to any suitable start pointfor answering a query. In examples, a current user context may bedefined by the query (e.g., by including a context keyword indicating asubject graph node to use as the start point). In other examples, acurrent user context may be an application-specific context (e.g.,“answering email”) suitable to determine a subject graph node to use asthe start point.

When encountering a node during the random walk (e.g., at the startpoint), the node may be examined to determine if it satisfies theconstraints of the query. If it does, it may be output in the subset ofuser-centric facts responsive to the query. Then, after encountering thenode, the random walk may continue, so that more nodes are encountered.To find more nodes, the random walk may continue along outgoing edges ofthe encountered node. Determining whether to continue along an outgoingedge may be a weighted random determination, including assessing aweight representing a likelihood of following the edge and samplingwhether to follow the edge based on the weight and random data, e.g., a“roulette wheel selection” algorithm implemented using a random numbergenerator. The weight of an edge connecting a subject graph node to anobject graph node may be determined based on the confidence value of thesubject graph node, the edge, and/or the object graph node. In anexample, the confidence values may be interpreted as indications ofrelevance to the user, so that edges which are more relevant or whichconnect more relevant nodes are more likely to be followed. The weightof the edge may be further determined based on other data of the subjectgraph node, object graph node, and edge, e.g., by assessing a relevanceof an edge to the query based on a natural language comparison of theedge type to one or more natural language features of the query.

By specifying constraints (e.g., features of the subject graph node,object graph node, and edge defining a user-centric fact), a user may beable to formulate a variety of queries to be answered using theuser-centric graph.

In addition to selecting a subset of user-centric facts satisfyingconstraints specified in a query, the user-centric graph may enableresponding to other specialized queries, e.g., slice queries and rankqueries.

In an example, a query is a slice query indicating a start node and adistance parameter. The answer to a slice query is a subset ofuser-centric facts including user-centric facts reached by starting atthe start node and traversing edges of the graph data structure to formpaths of length equal to at most the distance parameter away from thestart node. For example, if the distance parameter is set to 1, theanswer to the query will include the start node and all once-removednodes directly connected to the start node; and if the distanceparameter is set to 2, the answer to the query will include the startnode, all once-removed nodes, and all twice-removed nodes directlyconnected to at least one of the once-removed nodes. A slice query mayrepresent a collection of user-centric facts which are potentiallyrelevant to a particular user-centric fact of interest (e.g., auser-centric fact involving the start node), by way of being connectedto the start node by a path of at most the distance parameter. Bysetting a small distance parameter, the answer to the query mayrepresent a relatively small collection of facts that are closelyrelated to the start node; similarly, by setting a large distanceparameter, the answer may represent a large collection of facts that areindirectly related to the start node. As an alternative to specifying astart node, a query may also use a current user context as the startnode, thereby representing a collection of user-centric facts related tothe user's current context.

In another example, a query is a rank query to rank the plurality ofuser-centric facts based at least on a confidence value associated witheach user-centric fact, and the subset of user-centric facts is rankedin order according to the confidence value of each user-centric fact. Arank query may be interpreted as gathering user-centric facts which arelikely to be relevant to the user, without imposing additional specificconstraints on the query. In addition to ranking the plurality ofuser-centric facts based on a confidence value, the plurality ofuser-centric facts may be ranked based on other features. For example,user-centric facts may be weighted as more relevant if they are morerecent (e.g., according to a timestamp associated with each fact). Inanother example, a rank query may include a keyword and user-centricfacts may be weighted as more relevant if they include at least one nodehaving the keyword among its tags.

Although two examples of specialized queries (“slice” and “rank”) aredescribed above, a user-centric graph enables other kinds of specializedquery based on “slice” queries, “rank” queries, and other traversals ofthe user-centric graph according to any suitable constraints. Forexample, the subset of user-centric facts responsive to a slice querymay additionally be ranked by confidence value as in a rank query,thereby combining functionality of the two kinds of query. In anotherexample, a query is a pivot query indicating a start node. The answer toa pivot query is a subset of user-centric facts including user-centricfacts reached by starting at the start node and traversing edges of thegraph data structure to form paths of an unbounded (or arbitrary, large)length. A pivot query may be interpreted as a slice query that does notbound the length of paths reached by the start node, e.g., where thedistance parameter is infinite. In some examples, an answer to a querymay include a visualization of the answer subset of user-centric factsas a graph diagram, which may be annotated or animated to include anysuitable information of the user-centric facts (e.g., auxiliaryapplication-specific data indicated by a facet pointer of a nodeincluded in one of the user-centric facts).

A query may define a time constraint, so that a subset of user-centricfacts output in response to the query is restricted to user-centricfacts associated with timestamps indicating times within a range definedby the query. In some examples, time is an inherent property of nodesand edges in the graph data structure (e.g., each user-centric fact ofthe plurality of user-centric facts included in the graph data structureis associated with one or more timestamps). The one or more timestampsassociated with a node or edge may indicate a time when a user-centricfact was created, accessed, and/or modified (e.g., access metadata of anode defining the user-centric fact). Alternately or additionally, theone or more timestamps may indicate a time referred to in a user-centricfact (e.g., a time at which a meeting is scheduled, or any othertimestamp added by an enrichment adapter in the enrichment pipeline).The one or more timestamps may optionally be stored as searchable tags.

In some examples, the one or more constraints defined by a query includean answer type constraint, and accordingly, the subset of user-centricfacts selected responsive to the query may include only user-centricfacts that satisfy the answer type constraint. For example, an answertype constraint may constrain a feature of a subject graph node, objectgraph node, and/or edge of the user-centric fact. For example, an answertype constraint may indicate a particular type of subject and/or objectgraph node, such as: 1) either subject or object is a person; 2) bothsubject and object are coworkers; 3) subject is a place; 4) object is atopic; or 5) subject is a person and object is a scheduled event.Alternately or additionally, the answer type constraint may indicate oneor more particular subjects and/or objects, such as 1) subject is theuser; 2) subject is the user's boss, Alice; or 3) object is any ofAlice, Bob, or Charlie. Alternately or additionally, the answer typeconstraint may indicate one or more particular types of edge, e.g., byindicating a type of relationship such as a “sent email” relationship, a“went to lunch together” relationship, or a “researched topic”relationship.

In some examples, the one or more constraints defined by the query mayinclude a graph context constraint, and accordingly, the subset ofuser-centric facts selected responsive to the query may include onlyuser-centric facts that are related to a contextualizing user-centricfact in the user-centric graph that satisfies the graph contextconstraint. Two different user-centric facts may be described herein asrelated based on any suitable features of the graph data structure thatmay indicate a possible relationship. For example, when a graph contextconstraint indicates the user's boss, Alice, the contextualizinguser-centric fact may be any fact having a node representing Alice as asubject graph node or as an object graph node. Accordingly, the subsetof user-centric facts selected responsive to the query may include otheruser-centric facts having subject and/or object graph nodes that aredirectly connected, via an edge, to the node representing Alice.Alternately or additionally, the subset of user-centric facts mayinclude user-centric facts that are indirectly related to Alice, e.g., auser centric fact having a subject and/or object graph node that isindirectly connected, via a path of two or more edges, to the noderepresenting Alice. In some cases, a query including a graph contextconstraint may be a slice query, and the subset of user-centric factsmay include only user-centric facts that are related to thecontextualizing user-centric fact and reachable within at most aparticular distance of a node of the contextualizing user-centric fact.In other examples, a query including a graph context constraint may be arank query, and the subset of user-centric facts may include a selectionof user-centric facts that are most likely to be relevant to thecontextualizing user-centric fact, e.g., user-centric facts that areconnected to the contextualizing user-centric fact via many differentpaths, or via a path including edges having high confidence values.

Answering a query may include traversing the graph data structure basedon the one or more timestamps associated with each user-centric fact,which may be referred to herein as traversal of a time dimension of thegraph data structure. For example, answering a query may includestarting at a node associated with a time defined by the query, andtraversing the graph by following any edge with a timestamp indicating alater time, so that the timestamps increase in the same order as thetraversal. In other examples, answering a query may include traversingthe graph data structure by following any edge with a timestamppreceding a date defined by the query. Furthermore, the inherent timeproperty of each node and edge in the graph data structure may enable atimeline view of the graph data structure. In examples, an answer to aquery may include a timeline view of the graph data structure, e.g.,user-centric facts arranged in chronological order by a timestampassociated with each user-centric fact.

As another example of a specialized query, the graph data structure maybe configured to allow searching for a user-centric fact based on asearchable tag stored by the graph data structure for the user-centricfact. Searching for a user-centric fact based on a searchable tag mayinclude traversing the graph in any suitable manner (e.g., as describedabove with regards to slice queries or pivot queries), and whiletraversing the graph, outputting any user-centric facts encounteredduring the traversal for which the graph structure stores the searchabletag.

As another example of a specialized query, the graph data structure maybe configured to serve a user context query, by searching foruser-centric facts that may be relevant to a current context of a user.Accordingly, the user context query may include one or more constraintsrelated to the current context of the user. For example, the one or moreconstraints may include a time constraint based on a current time atwhich the user context query is served. Alternately or additionally, theone or more constraints may include a graph context constraint relatedto the current context of the user, e.g., a graph context constraintspecifying a task in which the user may be engaged.

In some examples, the one or more constraints of the user context querymay be based on state data of a computer service that issued the usercontext query. In some examples, the state data of the computer serviceincludes a natural language feature (e.g., an intent, entity, or topic),and the one or more constraints include an indication of the naturallanguage feature. For example, when the computer service is an emailprogram, the one or more constraints of a user context query mayinclude: 1) a time constraint based on a time at which a user beginscomposing an email; 2) a graph context constraint indicating a topic ofa subject of the email; and 3) a graph context constraint indicating arecipient of the email. Accordingly, a subset of user-centric factsselected responsive to the user context query may include user-centricfacts which are current (based on the timestamp) and which are likely tobe related to the user's task of composing the email (based on the topicand recipient).

Exemplary Computing System

The methods and processes described herein may be tied to a computingsystem of one or more computing devices. In particular, such methods andprocesses may be implemented as an executable computer-applicationprogram, a network-accessible computing service, anapplication-programming interface (API), a library, or a combination ofthe above and/or other compute resources.

FIG. 4 schematically shows a simplified representation of a computingsystem 400 configured to provide any to all of the compute functionalitydescribed herein. Computing system 400 may take the form of one or morepersonal computers, network-accessible server computers, tabletcomputers, home-entertainment computers, gaming devices, mobilecomputing devices, mobile communication devices (e.g., smart phone),virtual/augmented/mixed reality computing devices, wearable computingdevices, Internet of Things (loT) devices, embedded computing devices,and/or other computing devices. For example, computing system 400 mayinclude any combination of logic subsystems, storage subsystems, and/orother subsystems of one or more of user-centric graph server 103, clientcomputer 102, user-centric data sources 105A-105N, network 104, and/orother computers not shown in FIG. 1A.

Computing system 400 includes a logic subsystem 402 and a storagesubsystem 404. Computing system 400 may optionally include aninput/output subsystem 406 (e.g., comprising one or more input devicesor sensors, and one or more output devices such as a graphical displayand/or audio speakers), communication subsystem 408, and/or othersubsystems not shown in FIG. 4.

Logic subsystem 402 includes one or more physical devices configured toexecute instructions. For example, the logic subsystem may be configuredto execute instructions that are part of one or more applications,services, or other logical constructs. The logic subsystem may includeone or more hardware processors configured to execute softwareinstructions. Additionally or alternatively, the logic subsystem mayinclude one or more hardware or firmware devices configured to executehardware or firmware instructions. Processors of the logic subsystem maybe single-core or multi-core, and the instructions executed thereon maybe configured for sequential, parallel, and/or distributed processing.Individual components of the logic subsystem optionally may bedistributed among two or more separate devices, which may be remotelylocated and/or configured for coordinated processing. Aspects of thelogic subsystem may be virtualized and executed by remotely-accessible,networked computing devices configured in a cloud-computingconfiguration.

Storage subsystem 404 includes one or more physical devices configuredto temporarily and/or permanently hold computer information such as dataand instructions executable by the logic subsystem. When the storagesubsystem includes two or more devices, the devices may be collocatedand/or remotely located. Storage subsystem 404 may include volatile,nonvolatile, dynamic, static, read/write, read-only, random-access,sequential-access, location-addressable, file-addressable, and/orcontent-addressable devices. Storage subsystem 404 may include removableand/or built-in devices. When the logic subsystem executes instructions,the state of storage subsystem 404 may be transformed—e.g., to holddifferent data.

Aspects of logic subsystem 402 and storage subsystem 404 may beintegrated together into one or more hardware-logic components. Suchhardware-logic components may include program- and application-specificintegrated circuits (PASIC/ASICs), program- and application-specificstandard products (PSSP/ASSPs), system-on-a-chip (SOC), and complexprogrammable logic devices (CPLDs), for example.

The logic subsystem and the storage subsystem may cooperate toinstantiate one or more logic machines. As used herein, the term“machine” is used to collectively refer to hardware and any software,instructions, and/or other components cooperating with such hardware toprovide computer functionality. In other words, “machines” are neverabstract ideas and always have a tangible form. A machine may beinstantiated by a single computing device, or a machine may include twoor more sub-components instantiated by two or more different computingdevices. In some implementations a machine includes a local component(e.g., computer service) cooperating with a remote component (e.g.,cloud computing service). The software and/or other instructions thatgive a particular machine its functionality may optionally be saved asan unexecuted module on a suitable storage device. Non-limiting examplesof machines which may be instantiated by computing system 400 accordingto the present disclosure include browser 110 and user-centric graph300, as well as any computer services which may be configured tointeract with browser 110 and user-centric graph 300, e.g., an emailclient, a scheduling program, etc.

Machines according to the present disclosure may be implemented usingany suitable combination of state-of-the-art and/or future machinelearning (ML), artificial intelligence (AI), and/or natural languageprocessing (NLP) techniques. Non-limiting examples of techniques thatmay be incorporated in an implementation of one or more machines includesupport vector machines, multi-layer neural networks, convolutionalneural networks (e.g., including spatial convolutional networks forprocessing images and/or videos, temporal convolutional neural networksfor processing audio signals and/or natural language sentences, and/orany other suitable convolutional neural networks configured to convolveand pool features across one or more temporal and/or spatialdimensions), recurrent neural networks (e.g., long short-term memorynetworks), associative memories (e.g., lookup tables, hash tables, BloomFilters, Neural Turing Machine and/or Neural Random Access Memory), wordembedding models (e.g., GloVe or Word2Vec), unsupervised spatial and/orclustering methods (e.g., nearest neighbor algorithms, topological dataanalysis, and/or k-means clustering), graphical models (e.g., (hidden)Markov models, Markov random fields, (hidden) conditional random fields,and/or AI knowledge bases), and/or natural language processingtechniques (e.g., tokenization, stemming, constituency and/or dependencyparsing, and/or intent recognition, segmental models, and/orsuper-segmental models (e.g., hidden dynamic models)).

In some examples, the methods and processes described herein may beimplemented using one or more differentiable functions, wherein agradient of the differentiable functions may be calculated and/orestimated with regard to inputs and/or outputs of the differentiablefunctions (e.g., with regard to training data, and/or with regard to anobjective function). Such methods and processes may be at leastpartially determined by a set of trainable parameters. Accordingly, thetrainable parameters for a particular method or process may be adjustedthrough any suitable training procedure, in order to continually improvefunctioning of the method or process.

Non-limiting examples of training procedures for adjusting trainableparameters include supervised training (e.g., using gradient descent orany other suitable optimization method), zero-shot, few-shot,unsupervised learning methods (e.g., classification based on classesderived from unsupervised clustering methods), reinforcement learning(e.g., deep Q learning based on feedback) and/or generative adversarialneural network training methods, belief propagation, RANSAC (randomsample consensus), contextual bandit methods, maximum likelihoodmethods, and/or expectation maximization. In some examples, a pluralityof methods, processes, and/or components of systems described herein maybe trained simultaneously with regard to an objective function measuringperformance of collective functioning of the plurality of components(e.g., with regard to reinforcement feedback and/or with regard tolabelled training data). Simultaneously training the plurality ofmethods, processes, and/or components may improve such collectivefunctioning. In some examples, one or more methods, processes, and/orcomponents may be trained independently of other components (e.g.,offline training on historical data).

The methods and processes disclosed herein may be configured to giveusers and/or any other humans control over any private and/orpotentially sensitive data. Whenever data is stored, accessed, and/orprocessed, the data may be handled in accordance with privacy and/orsecurity standards. When user data is collected, users or otherstakeholders may designate how the data is to be used and/or stored.Whenever user data is collected for any purpose, the user owning thedata should be notified, and the user data should only be collected whenthe user provides affirmative consent. If data is to be collected, itcan and should be collected with the utmost respect for user privacy. Ifthe data is to be released for access by anyone other than the user orused for any decision-making process, the user's consent may becollected before using and/or releasing the data. Users may opt-inand/or opt-out of data collection at any time. After data has beencollected, users may issue a command to delete the data, and/or restrictaccess to the data. All potentially sensitive data optionally may beencrypted and/or, when feasible anonymized, to further protect userprivacy. Users may designate portions of data, metadata, orstatistics/results of processing data for release to other parties,e.g., for further processing. Data that is private and/or confidentialmay be kept completely private, e.g., only decrypted temporarily forprocessing, or only decrypted for processing on a user device andotherwise stored in encrypted form. Users may hold and controlencryption keys for the encrypted data. Alternately or additionally,users may designate a trusted third party to hold and control encryptionkeys for the encrypted data, e.g., so as to provide access to the datato the user according to a suitable authentication protocol.

When the methods and processes described herein incorporate ML and/or AIcomponents, the ML and/or AI components may make decisions based atleast partially on training of the components with regard to trainingdata. Accordingly, the ML and/or AI components can and should be trainedon diverse, representative datasets that include sufficient relevantdata for diverse users and/or populations of users. In particular,training data sets should be inclusive with regard to different humanindividuals and groups, so that as ML and/or AI components are trained,performance is improved with regard to the user experience of the usersand/or populations of users.

For example, a dialogue system according to the present disclosure maybe trained to interact with different populations of users, usinglanguage models that are trained to work well for those populationsbased on language, dialect, accent, and/or any other features ofspeaking style of the population.

ML and/or AI components may additionally be trained to make decisions soas to minimize potential bias towards human individuals and/or groups.For example, when AI systems are used to assess any qualitative and/orquantitative information about human individuals or groups, they may betrained so as to be invariant to differences between the individuals orgroups that are not intended to be measured by the qualitative and/orquantitative assessment, e.g., so that any decisions are not influencedin an unintended fashion by differences among individuals and groups.

ML and/or AI components can and should be designed to provide context asto how they operate as much as is possible, so that implementers of MLand/or AI systems can be accountable for decisions/assessments made bythe systems. For example, ML and/or AI systems should have replicablebehavior, e.g., when they make pseudo-random decisions, random seedsshould be used and recorded to enable replicating the decisions later.As another example, data used for training and/or testing ML and/or AIsystems should be curated and maintained to facilitate futureinvestigation of the behavior of the ML and/or AI systems with regard tothe data. Furthermore, ML and/or AI systems can and should becontinually monitored to identify potential bias, errors, and/orunintended outcomes.

When included, input/output subsystem 406 may be used to present avisual representation of data held by storage subsystem 404. This visualrepresentation may take the form of a graphical user interface (GUI).Input/output subsystem 406 may include one or more display devicesutilizing virtually any type of technology. In some implementations,display subsystem may include one or more virtual-, augmented-, or mixedreality displays.

When included, input/output subsystem may further comprise or interfacewith one or more input devices. An input device may include a sensordevice or a user input device. Examples of user input devices include akeyboard, mouse, touch screen, or game controller. In some embodiments,the input subsystem may comprise or interface with selected natural userinput (NUI) componentry. Such componentry may be integrated orperipheral, and the transduction and/or processing of input actions maybe handled on- or off-board. Example NUI componentry may include amicrophone for speech and/or voice recognition; an infrared, color,stereoscopic, and/or depth camera for machine vision and/or gesturerecognition; a head tracker, eye tracker, accelerometer, and/orgyroscope for motion detection and/or intent recognition.

When included, communication subsystem 408 may be configured tocommunicatively couple computing system 400 with one or more othercomputing devices. Communication subsystem 408 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. The communication subsystem may be configuredfor communication via personal-, local- and/or wide-area networks.

Language models may utilize vocabulary features to guidesampling/searching for words for recognition of speech. For example, alanguage model may be at least partially defined by a statisticaldistribution of words or other vocabulary features. For example, alanguage model may be defined by a statistical distribution of n-grams,defining transition probabilities between candidate words according tovocabulary statistics. The language model may be further based on anyother appropriate statistical features, and/or results of processing thestatistical features with one or more machine learning and/orstatistical algorithms (e.g., confidence values resulting from suchprocessing). In some examples, a statistical model may constrain whatwords may be recognized for an audio signal, e.g., based on anassumption that words in the audio signal come from a particularvocabulary.

Alternately or additionally, the language model may be based on one ormore neural networks previously trained to represent audio inputs andwords in a shared latent space, e.g., a vector space learned by one ormore audio and/or word models (e.g., wav2letter and/or word2vec).Accordingly, finding a candidate word may include searching the sharedlatent space based on a vector encoded by the audio model for an audioinput, in order to find a candidate word vector for decoding with theword model. The shared latent space may be utilized to assess, for oneor more candidate words, a confidence that the candidate word isfeatured in the speech audio.

The language model may be used in conjunction with an acoustical modelconfigured to assess, for a candidate word and an audio signal, aconfidence that the candidate word is included in speech audio in theaudio signal based on acoustical features of the word (e.g.,mel-frequency cepstral coefficients, formants, etc.). Optionally, insome examples, the language model may incorporate the acoustical model(e.g., assessment and/or training of the language model may be based onthe acoustical model). The acoustical model defines a mapping betweenacoustic signals and basic sound units such as phonemes, e.g., based onlabelled speech audio. The acoustical model may be based on any suitablecombination of state-of-the-art or future machine learning (ML) and/orartificial intelligence (AI) models, for example: deep neural networks(e.g., long short-term memory, temporal convolutional neural network,restricted Boltzmann machine, deep belief network), hidden Markov models(HMM), conditional random fields (CRF) and/or Markov random fields,Gaussian mixture models, and/or other graphical models (e.g., deepBayesian network). Audio signals to be processed with the acoustic modelmay be preprocessed in any suitable manner, e.g., encoding at anysuitable sampling rate, Fourier transform, band-pass filters, etc. Theacoustical model may be trained to recognize the mapping betweenacoustic signals and sound units based on training with labelled audiodata. For example, the acoustical model may be trained based on labelledaudio data comprising speech audio and corrected text, in order to learnthe mapping between the speech audio signals and sound units denoted bythe corrected text. Accordingly, the acoustical model may be continuallyimproved to improve its utility for correctly recognizing speech audio.

In some examples, in addition to statistical models, neural networks,and/or acoustical models, the language model may incorporate anysuitable graphical model, e.g., a hidden Markov model (HMM) or aconditional random field (CRF). The graphical model may utilizestatistical features (e.g., transition probabilities) and/or confidencevalues to determine a probability of recognizing a word, given thespeech audio and/or other words recognized so far. Accordingly, thegraphical model may utilize the statistical features, previously trainedmachine learning models, and/or acoustical models to define transitionprobabilities between states represented in the graphical model.

In an example, a method for providing contextual information to a userduring a browsing session, comprises: maintaining a user-centric graphincluding a plurality of user-centric facts associated with the user andderived from interaction by the user with a plurality of differentcomputer services; recognizing a context of interaction with a browserapplication by a user, the context defined by at least one or morecontextualizing user-centric facts in the user-centric graph; evaluatingthe context to identify assistive information pertaining to the context;and displaying the assistive information to the user. In this or anyother example, the assistive information is based at least on one ormore user-centric facts derived from interaction by the user with aplurality of different computer services. In this or any other example,the method further comprises recognizing a triggering fact in order totrigger recognition of the context; and adding the triggering fact tothe user-centric graph, wherein identifying the assistive information isbased at least on a relationship in the user-centric graph between thetriggering fact and one or more other user-centric facts in theuser-centric graph. In this or any other example, the triggering factincludes one or more previous interactions, by the user, with thebrowser. In this or any other example, the triggering fact includes aprevious interaction, by the user, with a computer service other thanthe browser. In this or any other example, the relationship in theuser-centric graph between the triggering fact and the one or more otheruser-centric facts in the user-centric graph is a temporal relationship.In this or any other example, recognizing the triggering fact includesassessing, for a candidate fact, a signal strength score indicating astrength of relationship between the candidate fact and user-centricfacts in the user-centric graph, and recognizing the triggering factresponsive to the signal strength score exceeding a threshold.

In an example, a for providing contextual information to a user during abrowsing session, comprises: maintaining a user-centric graph includinga plurality of user-centric facts associated with the user and derivedfrom interaction by the user with a plurality of different computerservices; recognizing a context of interaction with a browserapplication by a user; identifying assistive information pertaining tothe context, the assistive information being based at least on one ormore user-centric facts in the user-centric graph; and displaying theassistive information to the user. In this or any other example, thecontext is defined by at least one or more contextualizing user-centricfacts derived from interaction by the user with a plurality of differentcomputer services. In this or any other example, the method furthercomprises recognizing a triggering fact in order to trigger recognitionof the context, wherein identifying the assistive information is basedat least on a relationship in the user-centric graph between thetriggering fact and one or more other user-centric facts in theuser-centric graph. In this or any other example, the triggering factincludes one or more previous interactions, by the user, with thebrowser. In this or any other example, the triggering fact includes aprevious interaction, by the user, with a computer service other thanthe browser. In this or any other example, the relationship in theuser-centric graph between the triggering fact and the one or more otheruser-centric facts in the user-centric graph is a temporal relationship.In this or any other example, the triggering fact includes assessing,for a candidate fact, a signal strength score indicating a strength ofrelationship between the candidate fact and user-centric facts in theuser-centric graph, and recognizing the triggering fact responsive tothe signal strength score exceeding a threshold.

In an example, a system for providing contextual information to a userduring a browsing session, comprises: a logic device; and a storagedevice configured to hold: a user-centric graph including a plurality ofuser-centric facts associated with the user and derived from interactionby the user with a plurality of different computer services; andinstructions executable by the logic device to: recognize a triggeringfact in order to recognize a context of interaction with a browserapplication by a user; and identify assistive information pertaining tothe context, the assistive information being based on the context and atleast on one or more user-centric facts in the user-centric graph. Inthis or any other example, the instructions are further executable toreceive, from the browser, a computer-readable description of one ormore previous interactions, by the user, with the browser, and whereinthe triggering fact includes the computer-readable description of theone or more previous interactions. In this or any other example, thetriggering fact includes a description of a previous interaction, by theuser, with a computer service other than the browser. In this or anyother example, the relationship in the user-centric graph between thetriggering fact and the one or more other user-centric facts in theuser-centric graph is a temporal relationship. In this or any otherexample, recognizing the triggering fact includes assessing, for acandidate fact, a signal strength score indicating a strength ofrelationship between the candidate fact and user-centric facts in theuser-centric graph, and recognizing the triggering fact responsive tothe signal strength score exceeding a threshold. In this or any otherexample, the assistive information includes information based on one ormore of 1) emails to or from the user, 2) contacts of the user, 3)calendar data of the user, 4) documents, 5) web pages, 6) location dataassociated with the user, and/or 7) application usage data associatedwith the user.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

The invention claimed is:
 1. A method for providing contextualinformation to a user during a browsing session, comprising: maintaininga user-information graph including a plurality of facts associated withthe user and derived from interaction by the user with a plurality ofdifferent computer services, wherein the user-information graph includesapplication-specific constituent graph structures associated with theplurality of different computer services, each application-specificconstituent graph structure including edges connecting it to one or moreother of the application-specific constituent graph structures, eachapplication-specific constituent graph structure including a pluralityof nodes; recognizing, based on a relationship between theapplication-specific constituent graph structures, a context ofinteraction with a browser application by a user; evaluating the contextto identify information pertaining to the context, the informationincluding a natural language description; and displaying the informationto the user.
 2. The method of claim 1, wherein the information is basedat least on one or more facts derived from interaction by the user withthe plurality of different computer services.
 3. The method of claim 1,further comprising: recognizing a triggering fact in order to triggerrecognition of the context; and adding the triggering fact to the userinformation graph, wherein identifying the information is based at leaston a relationship in the user information graph between the triggeringfact and one or more other facts in the user information graph.
 4. Themethod of claim 3, wherein the triggering fact includes one or moreprevious interactions, by the user, with the browser application.
 5. Themethod of claim 3, wherein the triggering fact includes a previousinteraction, by the user, with a computer service other than the browserapplication.
 6. The method of claim 3, wherein the relationship in theuser-information graph between the triggering fact and the one or moreother facts in the user information graph is a temporal relationship. 7.The method of claim 3, wherein recognizing the triggering fact includesassessing, for a candidate fact, a signal strength score indicating astrength of relationship between the candidate fact and facts in theuser-information graph, and recognizing the triggering fact responsiveto the signal strength score exceeding a threshold.
 8. A method forproviding contextual information to a user during a browsing session,comprising: maintaining a user-information graph including a pluralityof facts associated with the user and derived from interaction by theuser with a plurality of different computer services, wherein theuser-information graph includes application-specific constituent graphstructures associated with the plurality of different computer services,each application-specific constituent graph structure including edgesconnecting it to one or more of the application-specific constituentgraph structures, each application-specific constituent graph structureincluding a plurality of nodes; recognizing, based on a relationshipbetween the application-specific constituent graph structures, a contextof interaction with a browser application by a user; identifyinginformation pertaining to the context, the information being based atleast on one or more facts in the user-information graph and including anatural language description; and displaying the information to theuser.
 9. The method of claim 8, wherein the context is defined by atleast one or more contextualizing facts derived from interaction by theuser with the plurality of different computer services.
 10. The methodof claim 8, further comprising recognizing a triggering fact in order totrigger recognition of the context, wherein identifying the informationis based at least on a relationship in the user information graphbetween the triggering fact and one or more other facts in theuser-information graph.
 11. The method of claim 10, wherein thetriggering fact includes one or more previous interactions, by the user,with the browser application.
 12. The method of claim 10, wherein thetriggering fact includes a previous interaction, by the user, with acomputer service other than the browser application.
 13. The method ofclaim 10, wherein the relationship in the user-information graph betweenthe triggering fact and the one or more other facts in the userinformation graph is a temporal relationship.
 14. The method of claim10, wherein recognizing the triggering fact includes assessing, for acandidate fact, a signal strength score indicating a strength ofrelationship between the candidate fact and facts in theuser-information graph, and recognizing the triggering fact responsiveto the signal strength score exceeding a threshold.
 15. A system forproviding contextual information to a user during a browsing session,comprising: a logic device; and a storage device configured to hold: auser-information graph including a plurality of facts associated withthe user and derived from interaction by the user with a plurality ofdifferent computer services, wherein the user-information graph includesapplication-specific constituent graph structures associated with theplurality of different computer services, each application-specificconstituent graph structure including edges connecting it to one or moreother of the application-specific constituent graph structures, eachapplication-specific constituent graph structure including a pluralityof nodes; and instructions executable by the logic device to: recognize,based on a relationship between the application-specific constituentgraph structures, a triggering fact in order to recognize a context ofinteraction with a browser application by a user; and identifyinformation pertaining to the context, the information being based onthe context and at least on one or more facts in the user-informationgraph, the information including a natural language description.
 16. Thesystem of claim 15, wherein the instructions are further executable toreceive, from the browser application, a computer-readable descriptionof one or more previous interactions, by the user, with the browserapplication, and wherein the triggering fact includes thecomputer-readable description of the one or more previous interactions.17. The system of claim 15, wherein the triggering fact includes adescription of a previous interaction, by the user, with a computerservice other than the browser application.
 18. The system of claim 15,wherein the relationship in the user-information graph between thetriggering fact and the one or more other user-information facts in theuser information graph is a temporal relationship.
 19. The method ofclaim 15, wherein recognizing the triggering fact includes assessing,for a candidate fact, a signal strength score indicating a strength ofrelationship between the candidate fact and facts in theuser-information graph, and recognizing the triggering fact responsiveto the signal strength score exceeding a threshold.
 20. The method ofclaim 15, wherein the information includes information based on one ormore of 1) emails to or from the user, 2) contacts of the user, 3)calendar data of the user, 4) documents, 5) web pages, 6) location dataassociated with the user, and/or 7) application usage data associatedwith the user.